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Abstract. We prove that the set of closed orbits in a real reductive representation contains a 
subset which is open with respect to the real Zariski topology if it has non-empty interior. In 
particular the set of closed orbits is dense. 



1. Introduction 

We consider the representation G x 1/ — > y of a real Lie group on a finite dimensional real 
vector space V. We say that the representation is real reductive, if there exists a complex 
reductive Lie group U'^ with Cartan involution 6 such that G is a ^-stable subgroup of U'^ 
with finitely many connected components and if the representation G x V ^ V extends to 
a holomorphic representation of C/^ on a complex vector space containing y as a G-invariant 
subspace. 

Denoting by Vc the set of closed G-orbits in V, our main result is the following. 

Theorem 1. Let G x V ^ V be a real reductive representation and W a subspace of V such 
that Vc OW has non-empty interior in W . Then Vc CiW contains a subset U of W which is 
open in W with respect to the real Zariski topology. In particular, VcOW is dense in W . IfW 
is G-invariant, then U can be chosen to be G-invariant. 

Theorem 1 is a generalization of a result of Birkes for representations of real forms. If G 
is a real form of C/*^ and if the G-representation V is given by restriction of a holomorphic 
representation of 1/"^ on V'^, then it is shown in [Bir71] that the closed G-orbits in V correspond 
to closed [/"^-orbits in V"^. We will give the usual proof using moment maps for this result. Using 
a stratification of V defined by orbit types of closed orbits, this enables us to prove Theorem 1 
for real forms. 

In our general situation, where G is not a real form, the idea is to complexify the G- 
representation V. If G is semisimple, G is a compatible real form of its universal complexification 
G^ and the G-representation V extends to a holomorphic G'^-representation V'^, so the proof 
for real forms applies. If G is commutative, the weight space decomposition can be used in order 
to give an explicit description of the set of closed orbits. 

The mixed case, where G is a product of its center and a semisimple subgroup, is more sophis- 
ticated. We can not describe the closed G-orbits in terms of the closed orbits of the semisimple 
subgroup and the center. Instead, we will construct a group H which contains G, which differs 
from G only by a subgroup of its center and which is a real form of a complex reductive group 
U'^. Moreover H admits a representation on V which extends the G-representation and which is 
given by restriction of a holomorphic [/'^-representation on V^. Thus Theorem 1 for real forms 
applies to the representation of H. Then it remains to establish a relation between the closed 
G-orbits and the closed i?-orbits. Our main tool is the fact, that the closed G-orbits inside the 
set of semistable points Sci-Mp) '■= {z G Z]G ■ z H /x^^(O) 7^ 0} with respect to a restricted 
moment map fip (see Section 3) are parametrized by the orbits of a maximal compact subgroup 
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of G in the zero-fiber ([HS07b], [Ki84], [Ne84]). In our situation a G-orbit is closed 

inside the set of semistable points with respect to a standard moment map if the corresponding 
H-oxh\l is closed inside the set of semistable points with respect to a shifted standard moment 
map. Since we do not know if the latter set of semistable points is dense, we give an equivalent 
formulation of Theorem 1 in projective space. Here the set of semistable points is dense by 
[HMOl]. 

It should be noted that in the special case where G is complex reductive and where its 
representation on V is holomorphic, the situation is much simpler. Here the strata defined 
with respect to orbit types of closed orbits are locally closed with respect to the complex Zariski 
topology. Then the assumption that the set of closed orbits has non-empty interior in W implies 
that lA can be chosen to be the intersection of a stratum with W . In particular all orbits which 
intersect lA are of the same orbit type. In the real case this is wrong. For this consider the adjoint 
representation of a real semisimple group. Here there exist finitely many Cartan subalgebras. 
An orbit through a regular element is closed and the isotropy equals the centralizer of the Cartan 
subalgebra. Therefore each Cartan subalgebra defines an open stratum which consists of closed 
orbits and the union of these open strata is dense. 

Theorem 1 may also be applied to actions on Kahlerian manifolds Z which admit a restricted 
moment map with respect to the action of G. If z G Z is contained in the zero fiber 
//^^(O), then there exists a geometric G-slice at z ([HS07b]). In particular the G-action in a 
neighborhood of z is described by the isotropy representation of Gz on the tangent space T^Z. 
Applying Theorem 1 to the isotropy representation, we get 

Corollary 1.1. Assume that there exists a restricted moment map fXp for the action of G on Z 
and that the topological Hilbert quotient SciMp) // G is connected. If the set of closed G-orbits 
inside the set of semistable points SciMp) has non-empty interior, then its interior is dense in 
Sg{Mp). 

An application of Theorem 1 is given in [St08]. If is a generic isotropy of the G-action on Z, 
define Y := {z £ Z;Gz = H and G ■ z is closed} and let Y denote the closure of Y. Let Ng{H) 
denote the normalizer of H in G. The main result in [StOS] is a description of the topological 
Hilbert quotient Z//G in terms of the quotient Yf/AfciH)- Here Theorem 1 applies to show that 
Y is smooth. 

The author would like to thank P. Heinzner for carefully reading the manuscript of this paper. 



2. Real reductive representations 

Let f7 be a compact Lie group and C/^ its complexification. We have a Cartan decomposition 
U'^ = C/exp(iu) where u is the Lie algebra of U. The corresponding Cartan involution 9 is given 
by ^(wexp(^)) = nexp(— ^). We call a closed real subgroup G of U'^ compatible, if it is invariant 
under the Cartan involution 6 and consists of only finitely many connected components. Equiv- 
alently, a Lie subgroup G is compatible if and only if G = exp(p) for a compact subgroup K of 
U and a subspace p of iu. The decomposition G = K exp(p) is called the Cartan decomposition 
of G. Note that in particular C/*^ is a compatible subgroup of f/'^ with Cartan decomposition 
U'^ = U exp(iu) where u denotes the Lie algebra of U. 

We call a representation G xV ^ V a real Lie group on a real vector space a real reductive 
representation, if G is a compatible subgroup of a complex reductive group U'^ and if F is a 
G-invariant real subspace of a holomorphic J7'^-representation space V. 
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3. SeMISTABILITY in PROJECTIVE SPACE 

The linear action U'^ xV —)■ V induces a holomorphic action of U'^ on the complex projective 
space ¥{V) such that the canonical projection ir: V\ {0} — > ¥{V) is f/'^-equivariant, i.e. com- 
mutes with the actions of U'^ on V and on F{V). Our goal is to give an equivalent formulation 
of Theorem 1 in terms of the induced G-action on projective space. For this, we describe the 
set of closed G-orbits in V in terms of a moment map on F(V^). 

We fix a Hermitian inner product on V such that U acts by unitary operators on V. Recall 
that the associated Fubini Study metric lj is a Kahlerian metric on P(t^), which is invariant 
under the action of the unitary group and which is in particular invariant under the action 
of U. The corresponding moment map /x: ¥{V) u* is defined by the equation d/U^ = i^,uj. 
Here = ^ exp(t,^) ■ x is the vectorfield induced by the action of the one-parameter group 

{exp(i^); t G M} and is the contraction of uj with Moreover, we require fi to be equivariant 
with respect to the coadjoint action of U on u* which is given by {g ■ (p){(,) = (^(Ad((7)^). 
Explicitly a moment map is given by ij,^{7t{v)) = i^^^^^. It is unique up to addition of an 
element in the dual of the center of u*. The moment map contains much information on the 
geometry of the [/'^-action on P(V^) ([Ki84], [Ne84]). Analogously the geometry of the G-action, 
which we are interested in, can be described in terms of the restriction of the moment map to 
(ip)* ([HS07b]). In order to simplify notation, we identify (ip)* with p. Then the restricted 
moment map /ip : F{V) — > p is given by /ip(x) = {fip{x),(,) = /i~'^(x) = 

For /3 G p, we define A4p{P) := fip^{(3) and in order to shorten notation we set A4p := fip^{0). 

Moreover we define 5g(A^p(/?)) := {x e F{V);G- x n Mp{f3) ^ 0} to be the set of G-orbits 
which intersect A^p(/3) in their closure. For /3 = we call Sci-Mp) '■= Sci-MpiO)) the set of 
semistahle points in ¥{V) with respect to /ip. If /? G p is contained in the center of u^, the 
shifted moment map /i + i/? is again a moment map and Mp{(3) is the zero-fiber of the restricted 
moment map /ip — j3, so Sg{Mp{(3)) is the set of semistable points with respect to /ip — j3. We 
have a relation between the closed orbits in Sg{M.p{(3)) and M.p{j3). 

Theorem 3.1 ([HS07b]). Let (3 G p be contained in the center ofu^. 

(1) A G-orbitG-x C SciMpiP)) is closed in SciMpiP)) if and only ifG-xnMp{p) + 0. 

(2) Every non-closed orbit in SciMpiP)) contains a unique closed orbit in its closure and 
has strictly larger dimension than that closed orbit. 

In particular, the closed G-orbits in the set of semistable points SciMp) are exactly those 
orbits which intersect the zero-fiber A^p = {n{v)] < ^>t:V,v >= for all ^ G p} of /tp. It is shown 
in [RS90] that a G-orbit G -v mV is closed if and only if it intersects M. := {v & V; < £,^v, v >= 
for all G p}. In our terminology, A4 is the zero-fiber of a restricted moment map on V for 
which the set of semistable points equals V. We observe that M \ {0} = 7r~^(A4p). Since vr is 
G-equi variant, together with Theorem 3.1 this proves 

Lemma 3.2. An orbit G ■ v, v ^ is closed in V if and only if G ■ tt{v) is closed in SciMp)- 

The set of closed G-orbits in V is C*-invariant. Therefore, applying Lemma 3.2, Theorem 1 
can be reformulated as follows. 

Let W be a linear subspace of V and assume that the intersection of the set of closed orbits 
in SciMp) with -ir{W) has non-empty interior in tt{W). Then it contains a subset U of T^iW) 
which is open in 'k{W) with respect to the real Zariski topology. If iriW) is G-invariant, then lA 
can be chosen to be G-invariant. 
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Now we give an explicit description of the set of semistable points. It is shown in [RS90] that 
each G-orbit in V contains exactly one closed G-orbit in its closure (compare Theorem 3.1). We 
define Af := {v £ V;0 & G ■ v} to he the nullcone of the G-representation V, i.e. the set of 
G-orbits which contain as the unique closed orbit in their closure. Note that M is C*-stable. 
By a result in [HS07a], the nullcone M is a real algebraic subset of V. Then Tr{M \ {0}) is real 
algebraic in ¥(y). 

Lemma 3.3. The set of semistable points is given by SciJ^p) = T^iV \ AA). In particular, it is 
open in ¥{V) with respect to the real Zariski topology. 

Proof Let x G SciMp) n tt{N' \ {0}) and y G G • x D Mp. Then y e Tr{M) since 7r(A/') is 
closed and G-invariant. But Aip = tt{A4 \ {0}) and Aif] J\f = {0}, a contradiction. This shows 

SG{Mp)C7T{V\J\f). ^ 

Conversely, iov v eV\J\f, there exists a w £ G ■ v n {M \ {0}). Then 'k{w) € G • tt{v) n Mp 
by continuity of vr and thus 7r(f) G SciMp)- □ 

There is no analogous statement for Sg{Mp{(3)). But if G is complex reductive, the following 
is known ([HMOl]). 

Proposition 3.4. Assume G = U'^. Then SciMpiP)) = 5[/c(Xiu(/3)) is open in F{V) with 
respect to the complex Zariski topology. 

4. Actions of real forms on projective space 

We call G a real form of f/^ if its Lie algebra is a real form of u*^ and if G intersects every 
connected component of U'^. If G is a real form of U'^ and if y is a real form of V, the sets 
of semistable points and the sets of closed orbits with respects to the actions of G and 11"^ are 
related. Observe that n{V \ {0}) C F{V) can be identified with the real projective space F{V) 
since V is totally real in V, i.e. V (liV = {0} holds. 

In order to shorten notation, we define S := S^c{Miu{P)) C ¥{V). We denote by Sg{Mp{P))c 
and Sc the set of closed G-orbits and the set of closed [/"^-orbits in iSg'(A^p(/3)) and S, respec- 
tively. The following proposition bases on a more general argument which states that the moment 
map /i and the restricted moment map Hp coincide up to a constant on iiT-stable Lagrangian 
submanifolds. 

Proposition 4.1. Let (3 £p be contained in the center of vP and assume that G is a real form 
of U'^ and that V is a real form of V. 

(1) SGiMpil3))nF{V)=SnF{V). 

(2) SGiMp{P))cn¥{V) = ScnW{V), i. e. a G-orbit G ■ x in 5g(Xp(/?)) n P(y) is closed if 
and only if U'^ ■ x is closed in S. 

Proof. Recall that we are given a [/-invariant Hermitian inner product (•, •) on V. By [RS90] 
this inner product can be chosen such that V is Lagrangian with respect to the Kahler structure 
which is given by its imaginary part. Since K operates on V, this gives {(,*v, v) = for v (z V 
and ^ G Therefore /x^ = for G t. Since G is a real form of U'^, we have a Lie algebra 
decomposition u = t © ip and we conclude A4p n F{V) = TWiu H F{V). Since (3 is contained in p 
this also gives Mp{(3) D F{V) = Mi^{(i) D F{V). 

For the first part of the proposition, the inclusion S C 5g(A^p(/3)) is shown in [HS07b]. 
Conversely, for x G Sg{Mp{(3)) n F{V), the closure G • x intersects Mp{(3) n F{V) = Miu{(3) n 
F{V). In particular, ■ x intersects Miu{P) n F{V) and thus x e S. 

For the second part, if G-x is closed in Sg{M.p{(3)), then it intersects A4p(/3) by Theorem 3.1. 
But since Mp{f3)r\F(V) = Miu{P)r\F(y), it follows that U'^-x is closed, again with Theorem 3.1. 
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Conversely, assume U ■ x is closed. The action of U is liolomorphic, G is a real form of ^7 and 
F{V) is totally real in P(V^). Therefore the real dimension of each G-orbit in U'^ ■xr]¥{V) equals 
the complex dimension of C/^ -x. In particular, all the G-orbits in the intersection have the same 
dimension. Moreover they are contained in SG{Mp{f3)) by the first part of the proposition. 
Then it follows from Theorem 3.1 that all these G-orbits, and in particular G • x, are closed. □ 

Remark. The same proof applies in order to show the result of Birkes ([Bir71]) that a G-orbit 
G-vmVis closed if and only if U'^ ■ v is closed. 

Let /? € p be contained in the center of and consider the shifted moment map fi + i/3. 
For its set of semistable points S C IP(V^), the GIT-Quotient vr: 5 — > S/jU'^ in the sense of 
Mumford ([MFK94]) exists and is an affine map. Each fiber of vr contains a unique closed orbit 
and every other orbit in the fiber contains the closed orbit in its closure. The quotient defines 
an orbit type stratification of S as follows ([Lu73],[Ri72],[LR79]). For a subgroup H of ?7^, we 
denote by S'^^^ := {x G S;U'^ ■ x closed, {U'^)x = H} the set of points on closed orbits, for 
which the isotropy group is given by H. Then the saturation Ih := 7r^^(7r(5^^^)) := {x G 
5; C/^ • x n S^^^ 7^ 0} is by definition the H -isotropy stratum. The closed orbits in Ih are 
exactly those which intersect S^^^ and they are the orbits of minimal dimension in I^. The 
isotropy groups of two points on an orbit are conjugate. Therefore Ih = IgHg-'^ foi^ 9 ^ U'^ ■ 
Since each orbit in S contains a closed orbit in its closure, the isotropy strata cover 5. Moreover, 
two strata Ih and Ih' are either disjoint or equal, where the latter is the case if and only if H 
and H' are conjugate. Considering conjugacy classes of isotropy groups, S is the disjoint union 
of the strata and the union is locally finite. A stratum Ih is open in its closure with respect to 
the complex Zariski topology and if the closure Ih intersects a stratum Ih' with Ih Ih' ) then 
after conjugation, H is contained in H', i.e. there exists a g' G C/^ such that gHg~^ < H' . This 
implies that for n G N the set Xn := Ucodim_f/>n -^-ff is open in S with respect to the complex 
Zariski topology. 

For n G N, we define On '■= {x G IP(V^); dim • x > n} to be the set of orbits with complex 
dimension at least n. Since the elements of the Lie algebra of C/*^ define holomorphic vectorfields 
on P(y), the set On is open with respect to the complex Zariski topology. 

Now, for actions of real forms, we prove a result which is slightly more general than the 
statement of Theorem 1 since we do not assume /? = 0. We need this generality in order to 
prove Theorem 1. Recall that 5g'(A^p(/3))c denotes the set of closed G-orbits in Sg{M.p{[3)). 

Proposition 4.2. Assume G is a real form of U'^ and V = V'^. let W be a subspace ofV and 
let n be maximal such that On H P(VF) is non-empty. Then, for a /3 G p which is contained in 
the center of vP , the intersection Sg{M.p{0))c H On nP(H^) is open in ¥{W) with respect to the 
real Zariski topology. 

Proof. For x G ^{W) C ¥{V), the orbit G • x is closed in SciMpifi)) if and only \i ■ x is 
closed in S (Proposition 3.4). By definition of n and On, an orbit U'^ ■ x with x G On nP(VF) is 
of complex dimension n. So for x G 5g(A1p(/9))c H On n P(VF) the orbit U'^ ■ x is closed and of 
dimension n. Then we have x € Ih where H is the isotropy group at x and since the dimension 
of U'^ -x equals the codimension of the isotropy group, U'^ -x is contained in the union of isotropy 

strata In = Ucodim H>n ■ 

Conversely, for x G In fi C„ n P(VF) the orbit f/*^ • x is of dimension n by the choice of n, so it 
is an orbit of minimal dimension in T„. Since the closure of U'^ ■ x is contained in X^, it follows 
from Theorem 3.1 that U'^ ■ x is closed. This shows 



SciMpmc nOnH Fiw) =innOnn F{W). 
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But Zn and On are open in T(y) with respect to the complex Zariski topology, so the intersection 
In n On n P(M^) is open in P(M^) with respect to the real Zariski topology. □ 

5. Proof of the main result 

We observed in Section 3 that for the proof of Theorem 1 it suffices to show that the intersec- 
tion Sg{Mp)c n tt{W) of the set of closed orbits inside the set of semistable points with -niW) 
contains a subset which is open in 'k{W) with respect to the real Zariski topology. In Section 4 
we proved a slightly more general result for actions of real forms. For the general case we will 
now give a construction of a representation of a real form which contains G in a convenient way. 
This enables us to prove Theorem 1 using Proposition 4.2. 

For our construction, we will make several assumptions on G which do not affect the set Vc 
of closed orbits. 

First, replacing G by its connected component of the identity G°, we may assume that G is 
connected. For this note that G° is compatible with Cartan decomposition G° = K° exp(p) and 
that a G-orbit G ■ v \s closed if and only \i G° ■ v is closed. 

The connected compatible subgroup G of admits a decomposition G = Gs ■ Gz, where 
Gs is a semisimple subgroup of G and Gz is the center. Here the Lie algebra of Gs is given by 
the Lie bracket [Q,g]. Both, Gs and Gz are compatible subgroups of C/*^. Let Gz = Kz exp(p^) 
denote the Cartan decomposition of the center and define Pz ■= exp{pz)- The group Gs • Pz is 
compatible in U'^ and an orbit of Gs ■ Pz is closed if and only if the corresponding orbit of G is 
closed. Therefore we may assume G = Gs ■ Pz- 

Dividing Pz by the ineffectivity of the P^-representation, we may furthermore assume that 
Pz acts effectively. Since pz acts by symmetric operators on V, we have a decomposition 

V = 0f=i Vi such that the action of the commutative group Pz on V is given by a real character 
Xi- Pz — on each Vi. By assumption Pz acts effectively, so these characters define an 
injective homomorphism x'- Pz (M^'')'^. Since Pz is contained in the center of G, each Vi 
is invariant under the action of Gs- The universal complexification G§ of Gs in the sense 
of [Ho65] is complex reductive, contains Gs as a compatible subgroup and by definition, it 
admits a holomorphic representation on V^*^, which extends the G5-representation. We define 
a representation of (C*)'^ on V'^ = 0f=i Vf" by letting the i-th component of (C*)'^ act on 
Vf' by multiplication. The actions of Gg and (C*)"^ commute, so we obtain a holomorphic 
representation of the complex reductive group G^ x (C*)*^ on V"^ . This group contains Gs x x{Pz) 
as a compatible subgroup. Therefore we may assume G = Gs x x{Pz), U'^ = G^ x (C*)*^ and 

V = V^. 

We define H := Gs x (M^'^)"'. Then H is a compatible real form of U'^ which contains G. We 
denote its Cartan decomposition hj H = Lexp(q). Note that by construction L = K and that 
q is the direct sum of p and a subspace 3 of the center of u*^. For an appropriate choice of the 
inner product, this sum can be assumed to be orthogonal. 

Altogether, for the proof of Theorem 1, we may assume that G is a compatible subgroup of 
a compatible real form H = ifexp(q) of C/^ and that H = G x exp(3). 

With these assumptions, we can complete the proof of Theorem 1. 

Proof of Theorem 1. We prove the equivalent formulation of Theorem 1 given in Section 3. 
Note that here tt(W) = P(VF) since V is totally real in V'^. Again, let n be maximal such that 
On n P(VF) / 0. Since On n W{W) is open in ^{W) with respect to the real Zariski topology 
and since Sg{.Mp)c H P(Ty) has non-empty interior in P(l^) by assumption, there exists an 
X e Sg{-M.^)c n On n P(T^). Let 5 G G with gx G M^. Then (5 := ^iq{gx) is contained in the 
orthogonal complement of p in q and in particular (5 is contained in the center of . 
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By Proposition 4.2 the set SH{Mq{P))c n n F{W) is Zariski open in F{W). It contains x 
since gx (z H ■ x Ci A4q{P). Therefore it is non-empty. Note that it is G-invariant if P(l^) is G- 
invariant. So for the proof of the theorem it now suffices to show that 5H(-A^q(/3))c C 5g(A^p)c- 

For this, let x G 5H(Alq(/3))c and y G -ff -xR A^q(/3). Then y e ^Ap since /3 is contained in the 
orthogonal complement of p. Consequently G ■ y is closed. The G-nullcone in V is //-invariant 
since H = G ■ exp(3) where exp(3) is contained in the center of C/^. Therefore SciMp) is H- 
invariant by Lemma 3.3 and we have x £ H ■ y C SciMp)- Let h S exp(3) with G ■ x = G ■ hy. 
Then G ■ x = h ■ G ■ y is closed in Sc{Mp) since G ■ y is closed. □ 
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